Often, it desired to interactively render 3D objects that have complex reflectance properties and elaborate surface details. Good examples are trees, grass, furry Teddy bears, feathers, and filled wineglasses. Because of their intricate shapes, and complex lighting properties, it is practically impossible to render such objects from parametric or geometric models such as polygon meshes. Therefore, models for these types of objects are usually constructed and rendered from high-resolution images acquired of the objects. This technique is generally known as image-based rendering.
Early image-based rendering methods required a very large set of images to achieve high rendering quality, see McMillan et al., “Plenoptic modeling: An image-based rendering system,” Computer Graphics, SIGGRAPH 95 Proceedings, pp. 39–46, 1995, Chen et al., “View interpolation for image synthesis,” Computer Graphics, SIGGRAPH 93 Proceedings, pp. 279–288, 1993, Levoy et al., “Light field rendering,” Computer Graphics, SIGGRAPH 96 Proceedings, pp. 31–42, 1996, and Gortler et al., “The lumigraph,” Computer Graphics, SIGGRAPH 96 Proceedings, pp. 43–54, 1996.
A more scaleable approach is to parameterize the object's images directly onto a surface of the model as textures. That approach, commonly referred to as surface light fields, enables more accurate interpolation between the images and has been employed by view-dependent texture mapping (VDTM), see Debevec et al., “Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach,” Computer Graphics, SIGGRAPH 96 Proceedings, pp. 11–20, 1996, Pulli et al., “View-based rendering: Visualizing real objects from scanned range and color data,” Eurographics Rendering Workshop 1997, pp. 23–34, 1997, and Debevec et al., “Efficient view-dependent image-based rendering with projective texture-mapping,” Proceedings of the 9th Eurographics Workshop on Rendering, pp. 105–116, 1998; surface light field rendering, see Miller et al., “Lazy decompression of surface light fields for precomputed global illumination,” Proceedings of the 9th Eurographics Workshop on Rendering, PP. 281–292, 1998, Wood et al., “Surface light fields for 3d photography,” Computer Graphics, SIGGRAPH 2000 Proceedings, PP. 287–296, 2000, and Nishino et al., “Eigen-texture method: Appearance compression based on 3D model,” Proc. of Computer Vision and Pattern Recognition, pp. 618–624, 1999; unstructured lumigraph rendering (ULR), see Buehler et al., “Unstructured lumigraph rendering,” Computer Graphics, SIGGRAPH 2001 Proceedings, pp. 425–432, 2001; and light field mapping (LFM), see Chen et al., “Light Field Mapping: Efficient Representation and Hardware Rendering of Surface Light Fields,” ACM Transactions on Graphics 21, Proceedings of ACM SIGGRAPH 2002, ISSN 0730-0301, pp. 447–456. 2002.
Surface light field approaches are limited by the complexity of the object's geometry. A dense mesh requires many vertices and leads to a decrease in rendering performance. Faster rendering with a coarse mesh leads to artifacts that are most noticeable at the silhouette of the object.
Silhouette clipping can improve the appearance of images rendered from coarse polygonal models. However, clipping is impractical for complex silhouette geometry such as trees, feathers, and furry textures. Concentric, semitransparent textured shells have been used to render hair and furry objects. However, to improve the appearance of object silhouettes, extra geometry called textured fins is used on all edges of the object. Neither of these methods uses view-dependent texture and opacity information from the acquired images.
Accurately silhouettes of high complexity but simple geometry can be rendered with opacity hulls and point models, see Matusik at al., “Image-based 3D photography using opacity hulls,” ACM Transaction on Graphics 21, 3 pp. 427–437, ISSN 0730-0301, Proceedings of ACM SIGGRAPH, 2002. Opacity hulls can use view-dependent transparancy values (alphas) for every surface point on the visual hull, see Laurentini, “The visual hull concept for silhouette-based image understanding,” PAMI 16, 2, pp. 150–162, 1994, or any other geometry that is bigger or equal to the geometry of the real object. Matusik et al. used opacity hulls with surface light fields, so called opacity light fields, for objects under a fixed illumination, and surface reflectance fields for objects under varying illumination.
Opacity light fields are effective for rendering objects with arbitrarily complex shape and materials from arbitrary view points. However, point models for opacity light fields are impracticably large, requiring on the order of 2–4 Gigabytes of memory, and it is difficult to accelerate point-based rendering method with graphics hardware. Consequently, it can take up to thirty seconds to render a frame, clearly not interactive. Therefore, polygon rendering methods that can be accelerated are preferred.
Therefore, it is desired to provide a system a method that can render models of complex textured objects from arbitrary views from a limited set of fixed images. Furthermore, it is desired that the method operates interactively, and uses a small amount of memory.